What Does an Image Look Like in Python?

Together with your partner, advance through this self-guided activity to learn more about how Python and PlantCV convert images into data.

What makes up images?

Digital images are made up of a mosaic of really small squares called pixels. You can think of them similar to how ancient art was made using mosaic tiles like the image below:

The arrangement of these tiles are fashioned in a way that it give the appearance of smooth lines that our eyes can follow. If the square tiles, or pixels, are too large, then it is much hard to make smooth edges and curves like 8-bit Mario.

"These small little dots [pixels] are what make up the images on computer displays, whether they are flat-screen (LCD) or tube (CRT) monitors. The screen is divided up into a matrix of thousands or even millions of pixels.Typically, you cannot see the individual pixels, because they are so small. This is a good thing, because most people prefer to look at smooth, clear images rather than blocky, "pixelated" ones.

However, if you set your monitor to a low resolution, such as 640x480 and look closely at your screen, you will may be able to see the individual pixels. As you may have guessed, a resolution of 640x480 is comprised of a matrix of 640 by 480 pixels, or 307,200 in all. That's a lot of little dots."

Images and Matrices

There are many ways we can store pixels, or the picture elements, of digital images. Take a look at the image of Felix the Cat below. We can represent this image with a 35 x 35 matrix using binary unites, or bits. These numbers will tell us the color of each pixel in the image, where 0 represents black and 1 indicates white.

So we see that these units exist within a pixel coordinate system that resembles a grid or matrix with rows and columns that each hold pixel values that represent a color on the digital image, with the origin in the top left corner and the coordinate values increase going down and right.

Furthermore, there are 8-bits in a byte, which is the fundamental unit of storage on computer. Since each bit contains one of two values and 8 bits make a byte, we find that there are 256 possible permutations of binary code that can represent each pixel. Binary images like the one above are just one way we can store pixel values. Each element in the matrix determines the intensity of the corresponding pixel.

However, when we look at grayscale images we see that the colors are not binary but instead exist within a gradient. With 256 possible bytes available, we see that the values in each coordinate become more discrete. We see that 0 still represents black (lowest intensity), but now white is represented by 255 (maximum intensity).

In Python, we can use image analyses packages to store our images as arrays using the NumPy package. Using NumPy we can determine the shape and size of the image, where the dimensions are ordered y (rows), x (columns, and z (channels) for images.

Color (RGB) Images

from IPython.display import Audio, Video, YouTubeVideo
id='l8_fZPHasdo'
YouTubeVideo(id=id, width="600", height="300")

• To submit code for execution, click on the cell to make it active, then hold SHIFT and press Enter

This is a color image encoded using a Red Green Blue (RGB) color model. Note: that in OpenCV the color order is BGR.

The color channels are stacked in a third dimension and increase from the 'front' to the 'back'

How is color broadcasted?

• It can be emitted from a black background (computer monitor).

• It can be reflected when deposited from a color ink jet onto white paper.

What is a color space?

Color information can be represented using different models, or color spaces. The color space describes the type of information used to produce representations of a color.

#What is the color of the grass outside? Print your answer.

Think about your response, what kind of information can you gather from that?

• Hue of the color (Green)
• Saturation of the color (how light or dark is the grass?)

The color we visualize is a combination of multiple colors but depending on the color space they could be a combination of Red/Green/Blue (RGB) or L* a* b* (CIELAB)

Let's take a look at TV and computer monitors. If you look very closely at your computer monitor (I DO NOT RECOMMEND DOING THIS), you may be able to see small patterns of circles that are combinations of little red, green, and blue lights. The intensity of these lights facilitate the display of the images on our monitor. How these colors mix together defines how "crisp" of a color and image we see.

Typical color images use an additive RGB color model:

Next, we will consider the CIELAB colorspace. Also referred to as the L* a* b* colorspace, this colorspace represents colors as ac ombination of lightness, A, and B components. Lightness represents the perceived brightness of the color, where A and B represent the green-red/magenta and blue-yellow color components, respectively. This colorspace is designed to approximate human vision better than the other colorspaces.

Let's play with our Jupyter environment to create some color images.

First, we need to load our libraries.

Example Workflow

Loading Libraries

# Matplotlib enables us to plot within the notebook, matplotlib is very powerful plotting library
%matplotlib inline 
# Imports NumPy package into notebook, essential for scientific computing
import numpy as np
# Imports PlantCV into notebook so that we can conduct plant phenotyping analyses
from plantcv import plantcv as pcv
# Imports library to handle workflow inputs compatible with parallel workflow execution.
from plantcv.parallel import WorkflowInputs
# Imports PyPlot which will provides us a MATLAB-like interface
from matplotlib import pyplot as plt

# Input/output options
args = WorkflowInputs(
    images=["Athaliana.png"],
    names="image",
    result="Athaliana_result",
    outdir=".",
    writeimg=True,
    debug="plot",
    )

For more information on the class Params, check out https://plantcv.readthedocs.io/en/4.x/params/

Setting Your PlantCV Environment Parameters

# Set debug to the global parameter 
pcv.params.debug = args.debug
# Change display settings
pcv.params.dpi = 100
#Now we set up our parameters for our PlantCV environment
pcv.params.text_size=20
pcv.params.text_thickness=20

Sample Plot

#Let's create a plot space that is filled with pixel values of zero (true black)
my_rgb_img = np.zeros((250,250,3), dtype=np.uint8)
pcv.plot_image(my_rgb_img)

You should see a black box that contains 250 rows and columns, filled with pixel values of [0,0,0]

#Now let's make a color image:
#We can give values to each color channel by defining the pixel in each RGB channel individually
#[:,:,0] -> Indexes the Blue channel
#[:,:,1] -> Indexes the Green channel
#[:,:,2] -> Indexes the Red channel

#Run the cell below to see how we can "turn on" certain pixels by indexing our NumPy array.
#We set the values of the subsetted data to their maximum value (255)
my_rgb_img[:150,:150,0] = 255
my_rgb_img[50:200,50:200,1] = 255
my_rgb_img[100:250,100:250,2] = 255
pcv.plot_image(my_rgb_img)

Notice how only a few of our pixels are "turned on" fully. By influencing the individual channels (the 3rd element in the tuple) and assigning those channels to pixel intensities between 0 and 255, we are able to alter the values of pixels discretely. Observe how in the region where all three channels are set to the max value (255) the color is white

Tinker with some of these values and see if you can create a mosaic of colors where different colors are represented within my_rgb_img.

Now, let's start working with some color images. We are going to begin by reading in a sample A. thaliana image.

Reading Images into PlantCV

# Read in the sample image of Arabidopsis into PlantCV.
# Leave **mode** as Default.
img, filepath, filename = pcv.readimage(filename=args.image)

Investigating Your Image

# Determine the shape and size of our RGB image below:
#The output will tell us (# of rows, # of columns, # of color channels)
img.shape

(2056, 2452, 3)

#Determine the data type of the image below
img.dtype

dtype('uint8')

#Identify the minimum pixel value found in the image between all three channels
np.min(img)

0

#Identify the maximum pixel value found in our image between all three channels
np.max(img)

255

#We described earlier that each color channel is its own matrix
#To show this, we will pull out the green channel and it will show each is a grayscale channel with its own respective data
#We will use pcv.plot_image(img=img[:,:,1]) to extract the green channel data
#[:,:,0] -> Blue channel
#[:,:,1] -> Green channel
#[:,:,2] -> Red channel
pcv.plot_image(img=img[:,:,1])

What is happening in the other channels?

• Change the 3rd element to 0 or 2 to see what the Blue and Red channels.

#Calculate the min, max, and mean values from the green channel to see how many pixels are represented in that channel
#We can check out the "pixel stats" for each channel to see where the most pixel intensity exists.
print(np.min(img[:,:,1]))
print(np.max(img[:,:,1]))
print(np.mean(img[:,:,1]))

0
255
78.90440266343364

Investigating Colorspaces

For image analysis and visual perception of color properties, other color models such as Hue, Saturation, and Value (HSV) or CIELAB (LAB) have advantages over RGB.

In the next exercise we are going to visualize the color spaces available in PlantCV so we can label the plant material and distinguish the plant from the background. We will use the function pcv.visualize.colorspaces() to help us see our image (img) in the various color models (HSV and LAB). We need to store the color space plot in a variable so just name the new variable cs. The last thing we will do is set original_img to False as we don't need to see our original image, we only care about the color space.

As you start typing out the method, be sure to use the TAB key to autocomplete the method so you don't end up with typographical errors. Once you complete typing pcv.visualize.colorspaces(), press SHIFT + TAB to view the helper to see how to set up the method.

# Visualize component HSV and LAB color spaces
cs = pcv.visualize.colorspaces(rgb_img=img, 
                               original_img=False
                              )

Let's take a moment to understand what each of the color models above are showing us. We will start with describing what HSV color spaces can tell us:

• Hue - refers to the color of the pixel, the absolute representation of the color
• Saturation - refers to how "colorful" the pixel is (i.e., the difference between light green and forest green)
• Value - refers to how "white" the pixel is (0-255)

• Lightness - similar to the Value color space, how much "white" or "brightness" is in the color
• A - This color space describes the green/red-magenta values
• B - This color spaces describes the blue/yellow values

We are visualizing our plant material in these spaces to maximize the differences between the "plant material" and the "background". In our image, these differences will be more easily observed. When we use use top view images, looking at our images in these color spaces are very important as they will help us discern "plant material" from "soil media"

print(input("Looking above at the 6 color spaces produced, which color space produces the greatest contrast between plant and background?"))

Use this color space that you determined to have the greatest contrast between plant and background, we are now going to convert our RGB image and cast it into grayscale but in the channel you chose.

(If you are unsure how to set up the method, press SHIFT + TAB to access the helper)

Convert Image to Grayscale

# The output of this method will be stored in the variable named *gray_img* since we are 
# producing a grayscale image.
# The functions we will be using are *pcv.rgb2gray_hsv()* or *pcv.rgb2gray_lab()* 
# Set the rgb_img to our *img* and define the channel with the color space you thought had 
# the greatest contrast.

# Convert the RGB image into a grayscale image by choosing one of the HSV or LAB channels
gray_img = pcv.rgb2gray_lab(rgb_img=img, 
                            channel="a"
                           )

Now that we have our grayscale image, let's see which pixels refer to our plant and which are the background

Visualizing pixel distribution in an image

Inputs:

• img = an RGB or grayscale image to analyze
• mask = binary mask, calculate histogram from masked area only (default=None)
• bins = divide the data into n evenly spaced bins (default=100)
• lower_bound = the lower bound of the bins (x-axis min value) (default=None)
• upper_bound = the upper bound of the bins (x-axis max value) (default=None)
• title = a custom title for the plot (default=None)
• hist_data = return the frequency distribution data if True (default=False)

Returns:

• fig_hist = histogram figure
• hist_df = dataframe with histogram data, with columns "pixel intensity" and "proportion of pixels (%)" Look at your picture, what percentage of the picture is plant versus background? It will be helpful to keep this in mind when you look at the histogram output.

# Use *pcv.visualize.histogram()* to see the distribution of pixel values in the grayscale
# and store it in the variable *hist*.
# We will create **50 bins** to start with but play with the bins until you can see 
# discrete peaks.
# Set *img* to *gray_img* since that is the image whose pixel distribution we wish to see.

# Visualize a histogram of the grayscale values to identify signal related to the plant 
# vs the background.
hist = pcv.visualize.histogram(img=gray_img, 
                               bins=50
                              )

What does your chart look like? Compare your histogram with your neighbor's and see where the peaks are on the chart. As a refresher, histograms aggregate information into discrete bins that satisfy a range of values. The more values that fall within that "bin", the larger the peak on the chart.

print(input("Where is the largest peak located on the histogram?"))

print(input("What part of our picture do you think is the big peak, is it the plant or the background?"))

Discuss amongst yourselves what the other peaks on the histogram are referring to in our image and change the number of bins in the histogram to see how the pixel intensities are sorted. How did your finds compare with what you just found out?

Creating a Mask

Binary Thresholding

The next step is to create a binary mask that excludes pixel data from the background, but shows the pixel intensities from the plant material. To do this we will use the function pcv.threshold.binary() to set a binary threshold that labels the plant pixels white and the background as black.

Inputs:

• gray_img = Grayscale image data
• threshold = Threshold value (0-255)
• object_type = "light" or "dark" (default: "light") - If object is lighter than the background then standard thresholding is done - If object is darker than the background then inverse thresholding is done

Returns:

• bin_img = Thresholded, binary image

(Press SHIFT + TAB key to see the helper so we can set the inputs.)

# We are going to store the information from this binary threshold in the variable bin_img.

# Use the histogram to set a binary threshold where the plant pixels will be labeled white 
# and the background will be labeled black
man_bin_img = pcv.threshold.binary(gray_img=gray_img, 
                                   threshold=100, 
                                   object_type="dark"
                                  )

Congratulations! You just set a manual threshold to exclude pixel data that you found does not represent the plant material.

PlantCV has the ability to automatically threshold using the function pcv.threshold.otsu() - our first step into using machine learning approaches.

(Use SHIFT + TAB to observe the helper and see how to set up this method, it isn't too much different than setting up a manual binary threshold.)

# Instead of setting a manual threshold, try an automatic threshold method such as Otsu
auto_bin_img = pcv.threshold.otsu(gray_img=gray_img, 
                                  object_type="dark"
                                 )

You might notice that some of the pixels from the background and color card made it into our Otsu-thresholded image, disregard these as they will not affect our analyses due on the design of our PlantCV workflow. What has happened is that the algorithm that Otsu's thresholding technique determined a pixel range that excluded the majority of the background, pot, and color card except for a few pixels (due to shading/color gradients).

Otsu's thresholding technique runs an algorithm with the following steps:

1. Process the input image
2. Obtain image histogram (distribution of pixels)
3. Compute the threshold value T
4. Replace image pixels into white in those regions, where saturation is greater than T and into the black in the opposite cases.

For more information on Otsu's thresholing technique, visit https://learnopencv.com/otsu-thresholding-with-opencv/

Next step is to create a region of interest (ROI) so that we can later quantify the phenotype of the plant. To do so, we will begin by creating a circular ROI using our original image as a reference. We will then overlay our binary mask so that PlantCV can pull the stats from it.

Region of Interest

We will use pcv.roi.circle() to superimpose a circle onto our image. Use SHIFT + TAB to open up the helper to see what inputs you will need.

Here is some helpful information that we will need to set up the code: Inputs:

• img = An RGB or grayscale image to plot the ROI on in debug mode.
• x = The x-coordinate of the center of the circle.
• y = The y-coordinate of the center of the circle.
• r = The radius of the circle.

Outputs:

• roi_contour = An ROI set of points (contour).

# Define a region of interest (ROI) where we expect to find a plant
# For this image, look at the image and try to determine where the origin
# of your ROI circle will be.
roi = pcv.roi.circle(img=img, 
                     x=1250, 
                     y=1020, 
                     r=700
                    )

Excellent work! Now we are going to use our binary mask image along with our ROI to create a filtered mask so that we can have PlantCV analyze the components of this plant and not the background or the color card.

We need to look at our manually-thresholded and the Otsu-generated binary masks to determine which of these binary masks contain the most "plant." This will be the binary mask that we use to further filter our mask for overlaying on our image for precise measurement.

In the previous step, we created an ROI circle that encapsulated as much as of the plant as possible. Since we care about total plant area, we want the binary mask that will contain the most plant material (i.e., the Otsu-generated binary mask - auto_bin_img).

Fixing Your ROI to the Binary Mask

We are going to use pcv.roi.filter() to refine our binary mask by selecting only pixels that are related to those pixels selected under the ROI. Recall that in both our manually- and Otsu-thresholded binary masks we had pixels from the chamber get extracted, using this function we can select only the relevant pixels in the plant. Let's investigate the requirements for this function with SHIFT + Tab:

Inputs:

• mask = binary thresholded image data to be filtered.
• roi = region of interest, an instance of the Object class output from a roi function
• roi_type = 'cutto', 'partial' (for partially inside, default), or 'largest' (keep only the largest contour)

Returns:

• filtered_mask = mask image.

*Work together to input the arguments to generate the filtered mask, set the roi_type to **cutto***

filtered_mask = pcv.roi.filter(mask=man_bin_img, 
                               roi=roi, 
                               roi_type="cutto"
                              )

Our mask looks great! Notice how we have eliminated all the pixels except those related to our plant, even those in the tip of the littlest leaf.

Our next step will be to gather object size data to see just how connected our mask is (i.e., is the mask one polygon or many little polygons close by).

Gathering Object Data

The function pcv.visualize.obj_sizes() will display all of the objects in the image using your binary mask and original image. Investigate the properties of this function and play around with the num_objects parameter to determine how many objects are in the image and the sizes of those objects. Inputs:

• img = RGB or grayscale image data
• mask = Binary mask made from filtered binary mask
• num_objects = Optional parameter to limit the number of objects that will get annotated

Returns:

• plotting_img = Plotting image with objects labeled by area

# Adjust the plot parameters so we can read the numbers.
pcv.params.text_size = 3
pcv.params.text_thickness = 5

# Set the number of objects to 1, our main object
sizes = pcv.visualize.obj_sizes(img=img, 
                                mask=filtered_mask, 
                                num_objects=1
                               )

There were 420 objects not annotated.

Well, apparently our filtered_mask isn't in one piece and is actually made up of about 421 objects.

Reducing Image Noise

We can connect some of the objects together using pcv.fill_holes(). This function fills all of the holes and connects several objects together with white pixels.

Inputs:

• bin_img = Filtered binary image data

Returns:

• filtered_img = image with objects filled

filled_mask = pcv.fill_holes(bin_img=filtered_mask)

Look at the image with mask overlay we produced, does it contain the aspects of the plant that we care about (i.e., as much "plant material" as possible)? If so, good! If not, then we need to become more discrete with our binary mask or we can filter the salt/noise so we get as much plant material as we can within our contoured layer.

Let's see how our flood fill worked using pcv.visualize.obj_sizes() again. We may not be able to completely connect all of the pixels, but we can try.

sizes = pcv.visualize.obj_sizes(img=img, 
                                mask=filled_mask, 
                                num_objects=1
                               ) 

There were 47 objects not annotated.

Looks as though there are still 47 more objects in this image, likely background.

Fill

We can use the Fill function to fill objects below a certain pixel size. This will help us create a single object for analysis.

Inputs:

• bin_img = Binary image data (use the
• size = minimum object area size in pixels (integer)

Returns:

• filtered_img = image with objects filled

clean_mask = pcv.fill(bin_img=filled_mask, 
                      size=400
                     )

sizes = pcv.visualize.obj_sizes(img=img, 
                                mask=clean_mask, 
                                num_objects=1
                               ) 

There were 0 objects not annotated.

Create labeled mask

We want to extract traits from each bean replicate, so we need to create a mask that has unique pixel values for each identified object.

Inputs:

• mask = cleaned mask image
• rois = (Optional) list of multiple ROIs (from roi.multi or roi.auto_grid)
• roi_type = (Optional)''partial' (for partially inside, default), cutto' (hard cut at boundary), 'largest' (keep only the largest contour)

Returns:

• mask = Labeled mask
• num_labels = Number of labeled objects

labeled_mask, num = pcv.create_labels(mask=clean_mask)

Gathering Information From Your Image

We're almost there! Now we just need to analyze our contours and see what information we can pull out!

Extract size data from your sample using pcv.analyze.size()

Inputs:

• img = RGB image data for plotting
• labeled_mask = Labeled mask of objects (32-bit).
• n_labels = Total number expected individual objects (default = 1).
• label = Optional label parameter, modifies the variable name of observations recorded (default = "default").

Returns:

• analysis_image = Diagnostic image showing measurements.

#shape_img = pcv.analyze.size(img=img, labeled_mask=labeled_mask, n_labels=num, label="plant1")
shape_img = pcv.analyze.size(img=img, 
                             labeled_mask=labeled_mask, 
                             n_labels = num,
                             label="Athaliana"
                            )

Extract color traits from your sample using pcv.analyze.color()

Inputs:

• img = RGB image for debugging
• labeled_mask = Grayscale mask with unique pixel value per object of interest
• n_labels = Total number expected individual objects (default = 1).
• colorspaces = 'all', 'rgb', 'lab', or 'hsv' (default = 'hsv')
• label = Modifies the variable name of observations recorded (default = "default").

Returns:

• analysis_image = histogram output

# Measure the color properties of the plant
pcv.params.debug = "plot"
color_hist = pcv.analyze.color(rgb_img=img, 
                               labeled_mask=labeled_mask, 
                               n_labels=num, 
                               colorspaces="hsv",
                               label="Athaliana"
                              )

Remember that the histogram data is going to tell us how many pixels of a particular value are in a colorspace. If we think about the various properties of plants (chloroplast density, photosynthetic efficiency, water content in leaves, etc.) and how they can be affect due to high heat environments, these measurements will be critical for our analysis.

Saving results

Analyzing a single plant is just the inital step of developing a workflow capable of performing high-throughput phenotyping. You will test your workflow on an increasing subset of image data to ensure its accuracy.

• A recommended subsetted dataset schedule for training your algorithm is 1 image > 5 images > 20 images > 50 images.
• Once you have developed sufficient accuracy with your workflow, then you will be ready to prepare a script for parallelization of the entirety of your image dataset for analysis.

# We will collect the data stored from *pcv.analyze.size()* and *pcv.analyze_color()* and save it as a Comma-Separated Values (CSV) files. 
# The filename will be set to *plant1_result*.
pcv.outputs.save_results(args.result, 
                         outformat="csv"
                        )

You can download the CSV file and view the attributes saved.

Congratulations, you have compelted this exercise! Please sit patiently as the rest of attendees complete their exercises.